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Axhe Development of the Mechanics of Film Rating J 



Ricki - The question we are talking about now is the mechanical 
procedure of rating films. Where did we start Last September - and try 
to figure this thing out and so before we had the marked paper we had 
what? 

Don - We had the Amidon-Flanders scale 

S - We started rating initially with the yellow sheets marked with 
numbers 

R - That^s what I thou^t 

S - And then we had trouble with the three seconds 
R - You know we started out with just a sheet of paper with numbered 
blanks on it and we tried to put a number from the Amidon Flanders 
scale into each blank every three seconds. 

S - That’s right 

R - And then as I remember the first problem we had was that we couldn’t 
do it . 

Don - And then Sheila made a tape where she called off every one, two 
and three seconds and that didn’t work either. 

S “ And then before that I used the stop watch too, remember? 

D - Oh, yes I 

S - And we stopped every three seconds for a while, too. 

P - We started at three and then we went to ten seconds . 

S - V^e tried stopping the projector and discussing what we had seen 
before . 

R - Then every three seconds was marked first with Sheila hitting the 
,9^ table every three seconds with a pencil and then with a number spoken 
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on a tape that ran the same time as the film. 

Lynn- And then the next thing that we did was to go ahead and rate it 
at the time of the reaction we had and then every ten seconds we stop- 
ped and talked about what we did. 

Ricki- Now let^s go back a little bit- why didn*t that first three 
second thing work? 

Pat- I think it was probably because of our- I think that probably now 

we ... 

Don- Well one of our problems was tiiat when the three second interval 
came around, what portion did you rate? what was going on during the 
three seconds or what went on at the time the thing happened. 

R- that was why we started getting Sheila to hit the pencil- to get 
us synchronized in rating. 

D- That^s right. 

R- The same second - so we coiald get reliability 

D- Then we decided to disregard what went on and rate every three seconds 
kinda interval jud^jhient and nobody liked that because sometimes nothing 
went on and it was confusing. We did miss one point- we changed from 
numbers to letters on the scale. We revised that back tiiere from the 
Amidon Flanders. 

R- By letters 

D- The categories one, two, three, four, and five to letters. 

R- Yea, a,b,c,d 

R- The reason I am asking this is that I realize that this is going to 
get transcribed by someone who doesn't know what we all know and we need 
to get real explicit now. 



-3- 



Don- We changed to letters which kinda stated category N for F for 
feeling D for direction and L for lecturing. 

R- and way back there on the three seconds what was confusing us? You 
remember, Beulah, we talked about it? 

R- I mean at first what kxnd of panic 

Everyone- Well, I couldn*t do it- it was terrible- well about a ten 
second interval was as long as we could keep it up 

R- Rjneiiber, at first we stopped every ten seconds and let everyone get 
their nerves back. 

Yea. 

Don- On the technical side, we decided then to design a machine which 
would move the paper forward and in a quick motion evei?y three seconds. 
Ricki- We worked on the sound thing first- our sound equipment - didn’t 
we? 

Don- Well, this was going on concurrently. It turned out that we gave 
up the thing before the machine ever got designed. Machine #1 . 

Ricki- I don’t remeniber machine #1. 

Don- I do. 

Pat- That first one rating we had back in the room. The meeting about 
the ratings , by Sheila ’ s desk and Judy ’ d' desk- I sat in on the last of 
it- this is where the first talk of some kind of machine was going on. 
Group- V/ith a rod all'.- the way through We talked about using the type- 
writer platens. And.-.allvkind* of ‘things, yeah.. 

Don- 1 was glad when we abandoned that 

Ricki- And v;hat was the problem that we were trying to solve there? 
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At that time what did we see as our problem in trying to rate? Well, 
of course, our frustration- we couldn^t do all of these MOTOR things 
and find our little mark. So what we were trying to solve here was 
to get the paper to take over some technical jobs we were trying to do. 
Lynn- And also like we were trying to count tdiree seconds between and 
we weren’t. 

Ricki- Yeah 

Lynn- Which is still our problem. 

Beulah- We found that thinking couldn’t be interrupted by a clang in 
the middle of it. 

Ricki- Yes, but I think that came later, though. I think that came 
after we had our sound equipment, because at first we were actually 
and then we decided to rate when the behavior changed. Way back at the 
beginning we were doing it righter than we were doing it for about three 
weeks in there. Before we ever tried to get time into it - three second 
intervals, we were giving a rating every time behavior changed. 

Don- Yeah, but we were doing that first on every ten seconds and we were 
averaging about three ratings for every ten seconds and then we decided 
that we could go on and do it like the Amidon Flanders thing and do it 
every three seconds. Which was about the same number anyway, and so .. 
Ricki- And Judy said that in the final learning session that we should 
practice it in its final form or you would never learn it. If three 
seconds was our goal we had to start doing it at three seconds from the 
very beginning. And so based on this theory we plunged into the three 
second thing and tried to gear everything else to this . 

? - and it didn't work? 
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Ricki- And it didn*t work? Wiy didn’t it work? It was the motor problem 
— - it was just simply the frustration, we couldn’t watch the films and 
look down at our papers and try to mark it on our papers at the same time 
and still watch the film. 

? - What’s the difference in the action going on in the different filna? 
Some of them would be slow action and some fast action and some would 
have a lot going on. 

Ricki- And what other problems did we find at that point? 

Don- Well, the one you pointed out has been solved mechanically, so 
that wasn’t the final reason for banning this. The final reason was 
that if you do it on a three second interval you have to decide exactly 
what you are going to rate. If you are going to make it one mark for 
every three seconds and three things go on in this three seconds then you 
have to decide which of these three things you were going to rate and 
this boiled down to what you would rate at that moment - what was going 
on at the moment and disregard everything else. 

Ricki- That’s right and what we were trying to do was cue in two differ- 
ent things. When that pencil hit we were supposed to make a rating, but 
what we found in fact was we were thinking of a rating and waiting for 
the pencil to hit and this was fine, but it wouldn’t come and then the 
action would change on the film and the pencil would hit and we still 
had this old rating in our minds of what had happened before and we 
couldn’t handle it cognitively. 

Group- Umhum 

Don- So then it was just about a month ago that you decided to go ahead 
and just rate it as it occured. 
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Lynn- And when we first did that, we felt like we had finally rated the 
film for the first time. 

Ricki- Yeah, luiihum, and what would you guess — — this took us about 
six weeks to start - four weeks. 

Don- about three weeks. 

Ricki- Three weeks! About the last week in October then we came up with 
this, huh? 

Don- It was four weeks ago. 

Ricki- Then it didn’t take us forever. 

Don- It was a couple of weeks that we actually struggled through 
Ricki- Is that all? It seemed like months. 

Dcii- And then we started out with the actual training on the basis of 
rating whenever you saw action and then moved it into training films. 
Remember? 

Ricki- Yeah and what we were doing was a - what we changed then was 
instead of a mechanical cue on our parts such as a called out number or 
pencil hitting the table the cue for response came from the film itself. 
A change of action. 

Don- A change of action. 

Ricki- Our field of attention didn't have to be changed. 

Don- Hum 

Ricki- Our direct field of attention was our cue for a response. 

Lynn- Is this where we had sound equipment? 

Ricki- No, I don't guess we had - when did we decide we had to have 
sound equipment? 

Pat- We decided that right at first - we decidcci when Dr. Holland was 
still in there. 
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Don- We couldn^t hear. 

Pat- And he was just in for a couple of sessions - I think it was about 
a week or so before we got it actually. 

Ricki- And so we found that sound . . . 

Don- Let's specify that by sound equipment we mean individual earphones. 
Ricki- Earphones. 

Don- For every rater. 

Jeanne- Also we were still clarifying the different feelings of interac- 
tion we finally decided that the category Work could stand for when 
the children were actually doing something. 

Ricki- Yeah, I think we are going to try to do that separately — talk 
about our changes in the rating scale. We are just trying to get the 
process now. 

Pat- We can do that as we .... on, Ricki. 

Lynn- WelJ., the volume on the headphones of the individual raters was 
the only thing we had trouble with. 

Ricki- And another thing we found that the headphones did was that they 
helped again to get at this central thing we were trying to do which 
was to make the experience a whole realistic one which put the sound in 
our heads. 

Lynn- They cut you off from everything else. 

Ricki- And allowed you to just be In the film and attend to it. 

Don- Especially with those particular earphones, I felt contact ri^t 
up to the cortex. 

A lot of laughter. 

Ricki- And they helped that way , you know. 
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Pat- I think something else that helped me too was having a room. Having 
a room where people don’t come in and out. 

Ricki- Where you can leave your equipment set up all the time. 

Pat- And it also gives us more light having that window: in there . 

Ricki- The light problem was a big one to try to solve. 

Don- There was another problem in there - we ruined one projector by 
running it back so much. 

Sheila- Yeah. 

Don-. When wg wore working out the rating scales. 

Ricki- And so then in training what we were doing was with our earphones, 
and with letters wh-inh helped us remember our scale a little better and 
also by this time we had all learned our scale which was kinda of an 
unadmitted problem at first, but none of us had really learned the scale. 
I m not sure that we could have though. 

Lynn- That wasn’t something we could have learned. 

Don- This came up , this scale, that we actually worked out was the 

Amidon Flanders, as a basis. But I feel that we actually went through 

a process of evolving a new scale and a method of rating. It’s brand 
new. 

Lynn- That’s a wonderful example of cognitive learning hooked up with 
SR response. Learning category labels and the " paper meaning” is one 
Sr response set to be formea and also an Sr response of marking a letter 
wheh you see an action on the film - but a higher level cognitive 
functioning intervenes in learning the p. 2 subtle meaninis of the cate- 
gories. This higher level can not be attained without having the experi- 
ence of seeing the film action. 
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Ricki - It was lika wa had to learn the scale on paper and then it 
was like we had to learn it all over again on rating films. 

Lynn - That^s really a high cognitive example of the SR basis... 
Ricki - Yeah, the way that I envision the paper that will be 
written out of this is an exampde of cognizant levels of cognitive 
behavior, huh? Fitting the task of what to these levels that go 
along. 

Lynn - But somebody could come along that hasn^t had it and say oh, 
no, that^s not the kind of cognizant function you had there. 

Jeanne - This cognizant assunption, I*m not sure exactly what that 
is , but it sure has called ny attention to my professors right now - 
their responses to what we do in class. 

Don - Sharpens your perception. 

Jeanne - Right; it sure has. 

Ricki - Now, what else was going... Oh, then what else we started 
doing was ten minute color film made in »60 and *61, and how long 
should it. ..it took us what... two weeks before we could even go 
through and rate a whole film. 

? - ...Before we could stand it. 

Ricki - Umhuh. . .before we could stand it. 

Don - Then at this point we started working on the kinds of machines 
which would move a piece of paper along at a constant rate of speed 
so that the person still wouldn*t have to look at the lines to make 
their ratings. That's a story in itself. 

Ricki - Unihum. 

Lynn - Shouldn't we say some tiling about overcoming our feelings? 
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Ricki - Yeah, to me this was the big thing that was going on. • .was 
the fear... or, I don’t know just what it was; I kept feeling it. had 
something to do withfrustration. .. 

Lynn - ...feel like you were about six steps removed from what you 
were doing. 

Ricki - Yeah. 

Lynn - You had to do the ratings of them, or they wouldn’t be done, 
but still they seemed so far away. 

Ricki - Well, how did we overcome that? I feel that it is overcome 
now, and I don’t really know how. 

Pat - I feel we have a pretty open group. You know... as far as... 
Don - Well, one thing I think that we could see was that the relia- 
bilities were going to run smooth enough to let us know that we were 
going to be able to rate films and get a meaningful scientific^ job 
of it. For a while there was a lot of doid)t as to whether it was 
going to be... you know, worth anything. 

Ricki - So then it was this other thing, too... making the scale 
really applicable to the task we were doing. That - closer to 
solving these mechanical problems. And this was done by becoming a 
real team, huh? All that talk that felt so frustrating really had 
a lot to do with our beginning to feel that we could do this job. 

Pat - You know what I felt at first? I never really felt that it 
couldn’t be rated. I kinda felt like. ..well, like I just wanted it 
to be done like I wanted it to be done, you know? 

Ricki - Yeah. 

Pat - It would be all right. . .and then what I thixiik we arrived at 
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with everyone talking and expressing themselves is a content attitude, 
and I think that maybe, even if someone takes our manual, this might 
be a scale that may not be an absolute thing. 

Lynn - I think that everyone would want their own manual. I feel 
like it is only today that we have a real positive feeling toward 
rating films. 

Ricki « Well, remember the day that we did the first. ..the film of 
the "starfish” girl? That was the day we began to feel... and I think 
it helped she was a good teacher. I don't know why it was a hard one 
to rate in a way. 

? - ...so much happened. 

Lynn - She was the first really good teacher we had seen. 

Pat - Yeah. 

Ricki - So we would recommend that, huh? 

Pat - Yeah, I think it would make you feel better with your rating 
if you see a good teacher; good in 'die sense that there is a lot 
of... maybe there's nothing bad going on... she wouldn't be a good 
therapist or anything, it's just that a lot of activity and smiling 
people and everything. . .you know. . . 

Ricki « Well, could it be that we relate so closely to the film with 
this close sound equipment and 'the dark room and everything that if 
we are watching someone who has a positive regard for the students 
she is working wi'tii at this point, it might somehow generalise it 
to us? 

Lynn - Yeah, that could have been, 

Ricki - ...and liiat somehow we become better students just like the 
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children in the film do. 

Pat - I don’t know... 

Ricki - ...because we sure had some frustration with bad teachers. 

Pat - But I don’t know whether it’s that... it’s more like we’re 
frustrated, and we see a situation that we think is bad, cuid we’ve 
got our bad feeling about what we’re doing and we*ve got this lousy 
QcxinQ uu, and everything is just bad. 

Ricki - Well, I think just the opposite of this, Pate I mean the 
other side of tdiis coin; the same coin, but it’s the good things go 
on, we feel good... if bad things go on in the film, we feel frustrated, 
Lynn — If this had not come at a •. . .maybe this came at a pretty 
crucial point, rather than... maybe if we had started off with good 
films, maybe build up some kind of negative response to films, but 
we would view with frustration even the good films; whereas, the 
good films in the latter part can be reinforcing. I wonder. 

Don - Well, it’s interesting, because I thought on some of the 
training films, the color films - they were generally good, and 
there wasn’t too much bad stuff in that. You are talking now about 
when we got introduced to the black and white. 

Ricki - No, this was a color film. The ’’starfish girl” v;as a good 
color film. 

Sheila:- She was preceeded by the person with the beads. The girl 
with the coral. 

Ricki - Well, the ’’starfish girl” was... we felt was a good teacher. 
Jeanne - Well, we had one criticism of her. She called coral 
’’seaweed” . 
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2<on - This was technically Incorrect, and we used this for a TAF 
rating, and we are getting way off content. 

Ricki - Yeah. 

Don - Right now, we used this in a TAF rating in a TAF team, but she 
was so content incorrect to me, she was bubbling and running a ll 
around the place, but everything she was saying was wrong. 

? - Well, I guess for the rating was what we were rating was her posi- 
tive... well, I guess as far as rating goes, 1 guess what we were 
responding to was her positive regard. 

Don - Regard... 

Sheila - I thought of something else that helped, too. When we divided 
up into teams, that helped an awful lot, and tiien something else 1 
don’t think I would have a very positive feeling about this if I 
didn’t like the other raters. 

Ricki - Yeah, 1 think liiatis what we have been saying in a lot of 
ways. 

Sheila - If I didn’t like the other raters and they said something 
that disagreed with me, I wouldn’t like them, and I wouldn’t be 
happy about it. 

Pat - Is this saying that if you have a very hostile feeling, 
covertly, toward another rater, don’t rate because your reliabilities 
could go way down. 

Ricki - Unihum. 

Pat - Well, then, this might be an important criterion. 

? - Well, all the things we do... 

Sheila - Well, if you just don’t basically like each other as a group 
you might just as well forget it. 
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Pat: Then how could you take some of the things that we have said 

because we have disagreed with each other so vehemently at times that 
if there was a basic dislike there would be a really - 
Don - Ye^, I remember the time that I got the most - really felt the 
most enthusiastic and bubbly and all that stuff was when we did go to 
work on TAF’s and come out with fantastically high inter- reliability 
on about the second go-round you know, (everyone) And, I don^t know, 
maybe that is where I am trying to get the TAF into here because it 
seemed to come along pretty much after the point where we started rating 
with the moving paper which ended up not working, but when we had to 
abandon that thing and start working on tlie final draft of the machine, 
then didn^t both teams start working on the TAF? 

Sheila - No, we worked on - don^t you remember at the very first meeting 
she said we are going to do three things to rate the films, the TAF, the 
Ryans scale and the Ami don -Flanders and we decided which order they 
would be in and the Amidon, the TAF, and the hot-warm, the hot-cold 
thing and then we just all tackled the Amidon-Flanders and when we 
began to start and get familiar with that, that was when we went on 
to the TAF . That has only been recently . 

Don - You have only had one session. 

Sheila - Well, our team, anyway has just been recent with the TAF. 

Rickx - Yeah, well, we have only been working on the TAF for the last, 
what, ten days. But I think when we got to feeling real good was when 
we made the film for the demenstration and we got to try what we thought 
was our reliability was real high so we wanted to share this and see if 
it would hold up with everybody there - so we saw the same ten minute 
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film three times and agreed on a rating for the whole film on Amidon 
Flanders. And then we presented it on that Friday morning and it was at 
that point that we really thought we could rate films. 

Pat- I feel that probably still if we work out the scaile then we will 
be 

Pat- You know not.... 

Ricki- And then from there we started - well started with Jeanne ^d Pat 
some , too . 

Pat- 1 missed a week 
Ricki- Yeah 

Pat- Came back when you had switched 

Ricki- Yeah, uh, huh, and Pat was the first one we tried it with and 
Jeanne the second - to try to teach the whole business we had been 
through. By just having someone sit down and copy the letters, the 
ratings as they were spoken to the film and not try to think at all 
thomsolves and I think that Pat thought she caught up with a week’s 
work in what- just an hour or so- and helped that way and how 
did you feel about it, Jeanne? 

Jeanne- I liked it, I felt reeil good about it. I didn't know what I had 
walked into when I first got there, but it felt natural , felt real 
natural. 

Ricki- I think this is something else we want to do- get that thing 
really fixed and may be on a film- I have been thinking that maybe on 
a film have a wider range of behavior than on feedback. The scale- and 
we changed our scale some - seems to me that tiiis is something we ought 
to do- take one of those color films that has a wider range . I don't 
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think we did it on the starfish. 

? - Umhum. 

Ricki - Did we do it on our... 

Sheila - I gave you a demonstration on the starfish, remember when 
we were going to rate that film again, I said I can’t, I have seen it 
too many times; I don’t want to see it anymore. The Starfish Film... 
I think that’s the one I demonstrated with. 

Don - Well, the purpose of the tape we have run over something that 
we got to go back and fill in a little bit. because one thing that 
you decided to do was... well, it was decided, anyway... to have seg- 
ments spliced into this training tape every thirty seconds... or 
minutes? 

Ricki - Into the training film. . . 

Don - ...the training film. Every thirty seconds was it?^ 

Ricki - A number. . . 

Don - Yeah. 

Ricki - That’s right. 

Don - In this way when you were training, you could do thirty seconds 
and let everybody put their rating down or call them out, and thei 
if there were any questions you could go back from that point or 
later on you could back up to that particular number, and the 
training film was to be broken down into numbered segments so that 
you could stop and give people a breather and check out things and 
answer any questions and then go ahead. 

Ricki - This came up because somebody would say, ”I disagree here,” 
and we wouldn’t know where "here” was on the film. We would know 
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where it was on our rating scales, and we would have to run b ac k 
through the film and work the film out and break it and look all 
over the place. 

Jeanne - The films are spliced now in thirty minute segments? 

Don - Andy was going to do one like that. 

Ricki - We just didn^t do it. We talked about it and thought it 
would be good and just didn^t do it. Another thing we did in train- 
ing was we would take turns in calling out. . .one person would call 
out their ratings of the film as we went along. 

Lynn - That was one of the most helpful thdLngs to me. 

Ricki - Most helpful for me, too. Then we would stop after Idiirty 

seconds of this and someone else would take over and call out their 
rating. 

Lynn - You know what that does? It makes you more personally commit- 
ted to rating yourself, too. 

Ricki - Yeah. 

Lynn - You stop just sitting there... 

Ricki - You stop confronting your paper. . .yeah. 

Lynn - It^s not nearly so boring... 

Ricki - . . .not nearly so what? 

Lynn - ...so boring. 

Ricki - Yeah, and defeated. 

Jeanne - You know, I hope we don^t splice the numbers through because 

it would be as visually disturbing as the pencil was audibly disturb- 
ing. 

Ricki - Well, these numbers were just for demonstration films where 
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we would be (.training people. 

Don - It wouldn’t be for all of them. 

Ricki - Unihum. 

Don - strictly for the training films. 

Jeanne - Well, you know when Don runs the training films, he just 
turns the film off and we discuss the thirty seconds. 

Ricki - Yeah, we did that, too. 

Don - What Ricki is getting at is that somebody would say, "Well, back 
there that teacher said," so and so; and that’s what we ran into, and 
you keep trying to figure out where "back Idiere” was, and you can’t; 
and this didn’t happen when people were really trying to understand 
what the differences stood for what the little nuance question and 
directions and directions for them and things like that. And we had 
to back up and try to find it, and the numbers would do that, and 
once you had all of these things zeroed in and then you wouldn’t need 
them anymore. 

Sheila - We also ruined a projector that way, too. 

Don - Yeah. 

Ricki - So then, we feel that there are two things that are essential. 
Well, three things are essential for training, one is enough discus- 
sion of what is going on and a sharing of feeling among the members 
of the team so that everyone begins to feel that he can disagree 
at any time about anything. That there is nothing too small or too 
big that you can’t disagree about it. So this is some kind of 
rapport which we have decided came through our ability to learn to disa- 
gree with each otiier, huh? Somewhat... 
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Lynn - I had to admit that 1 felt frustrated. . .sitting there feeling 
guilty about it. 

Ricki - Yeah, and we can’t... 

Sheila - Yeah, like everyone else is all excited, and I think it*s 
horrible. 

Everyone - Unihum. 

Ricki - And then another thing is the training film. That is, trying 
it by rote... just sitting down and copying it by numbers - I mean, 
letters - spoken so you have at least done it successfully, even 
though you weren’t thinking while you did it. It gives you a sense 
that it can be done; sind then, another thing was speaking out your 
response aloud to the team as you are rating and then stopping and 
talking about those and sort of building up reliability in thirty 
second segments in that way. 

Pat - Another thing we did was to mark and... our people feeling free 
at any time to say ’’stop”... you know, the film. ..right now. 

Ricki - That took a while in itself, didn’t iti It seemed to me 
that everyone was afraid to say ’’stop,” so we would all go beyond 
our endurance. 

General laughter... 

Lynn - I think that sort of stuff is really kind of inportant; it kind 
of previews you to write the film... You know, you can get our;.. 
Laughter... 

Don - Well . . . 

Ricki - Have we caught everything up to where we got our machines? Can 
you thing of anything, Sheila? 
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Sheila - Only one thing I can think of. Didn’t you say that if a 
person would call out their ratings aloud with the team, it would 
influence other ntembers? It seems like I remember you £»aying oxiC'9- * 
that: you^ would call these out as we were going along, and Pat said 
what I was thinking, that... well, it would be much easier to put down 
what you said... your ”D,” rather than ray ”L.” 

Pat - This was when we were thinking about doing this as a permanent 
thing, instead of writing...! would tend to put down what someone told 
me to put down. We ought to make it very clear that this is for 
training purposes right at first. 

Sheila - Yeah, and you could not continue very long with this; not with 
any individual. 

Ricki - No, no, oh no... and that this be traded off. 

Pat - That’s right. 

Ricki - I feel like I did too much of it there for a while. 

Pat - Well, at first you were doing all of it, but we were at a kind 
of a different idea there. We started practicing, and then I think 
we started getting more relaxed, too...l think at first that no one 
wanted to take the responsibility of... you know, of calling their own 
numbers out. I know I didn’t. I didn’t mind writing them down, but 
I didn’t want anyone to ever knov/... 

General laughter... 

Lynn - That’s the first time that I ever really felt like anybody 
agreed with me because before. •• it was like we agreed, but there 
were things in between...! mean, we might have a ”Q” and an ”E”, but 
we had six letters in between; but here, the calling it and we agreed 
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makes you feel... 

Ricki > Then what we are saying is that in training it is real 
important to get reinforcement wherever and in any way you caUj be- 
cause that really is pretty insurmountable when you start out. 

Lynn - 'Made me think it wasn't going to work. 

Ricki - I really felt like Lynn. I really felt that we weren't going 
to be able to do it when we first. . .well, I would feel that it would 
one day, and then, the next... well, it just couldn't be done, this 
three second stuff; I can rate thirty seconds of film; I can't rate 
fifteen minutes of film. 

Beulah - Even when 1 would come in. ..always behind because of my 
schedule...! could see so much progress and it would make more 
sense to me, and that maybe I would be able to do it, too. ..that 
I would be just like this, and you all would be all so disgusted 
because you just kind of had lived with it, and I could kind of retreat, 
and I could see this progress, and I was so proud... when 1 had gotten 
here, and I would say, "I can do this someday." I could see your 
progress every time X came. 

Ricki > We would see it in moments, and then we would. . .well, the 
categories were giving us trouble, too. 

? - Yeah. 

Sheila - We just now got our separate ideas straightened out. Not 
more than a week and a half ago. 

Ricki - Yeah. 

Sheila - One thing about two teams.. .they were once elementary and 
secondary, and they are no longer elementary and secondary; they are 
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just teams. 

Rlc*ki - Yeah, tjiat was another thing, too. And why did the dividing 
of the teams help, too, Sheila? 

Sheila - You could belong to a little group. . . 

Sheila - Even now, I say, ^Secondary team is meeting in the afternoon”; 
you know, ”I belong to secondary team; I don’t belong to elementary 
team." Hal You know. ..it isn't reedly that bad as it sounds, it’s just, 
you know. 

Ricki - Yeah. 

Sheila: You had your bunch of people and you know what they thought 

and you felt secure in little V-Hall - you know, there with your little 
team. I did. 

Lynn - Didn’t you feel that you had to be there all the time? 

Ricki - Yeah, we found that our sessions were running too long - a three 
hour session at one time was inpossible. 

Lynn - That was one thing that we just agreed among ourselves that ought 
to go down. 

Ricki - A two hour session is maximum and if you have acconplished 
nothing, then you still quit. 

Sheila - That would make me feel good too - you know, let’s quit after 
so long. 

Ricki - I thought that that was the time that Lynn felt like she could 
somehow join the group, I never knew why, but - 

Lynn - Well, I think it was hating to have my Friday afternoon’s taken 
up. 

General laughter 
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Ricki - Now, we have just incorporated that, if we don’t announce 
when a meeting is starting, we absolutely announce when it is ending 
and we keep that ending time if it is in the middle of a film and we 
' don’t start tc film df it would run us over. And so this has something 
to do with respectii^ our weaknesses - our attention span just can’t 
last for the length of time it would just have to last for a very long 
period of time. 

So we are up to where we are now 

Don - Yeah, perhaps you ought to put in there why we had to abandon 
the machines. 

Lynn -• You might want to describe the machines. 

Sheila - Have a picture 

Don - We ended up with four separate machines with tapes which ran 
at four separate rates of speed and we had a switch arranged so that 
when you switched on the projector you could switch on these things 
simultaneously and it turned out that there was no way to get these 
things exactly accurate and so that meant that we had to design a 
machine which somehow or other could afford perfect synchronization 
eind grader ratings because just the slightest little bit of difference 
would throw the whole thing off and Idiat’s why we quit and haven’t been 
doing anything for a week. 

? - Well, has Jim Felts come up with a machine that 

Don - He’s working on it, yeah - it’s real interesting. This was an 

accidental product of one of Frances’s. 

Ricki - It was an idea you had way back at the beginning, Don, to run 
the sheets of paper across 

Don - No, listen, Frances had this idea - I thought the possibility of 
o 
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trying four machines together so that we would have one drive and it 
turned out that we got four machines liiat turned exactly 1 rpm and 
they were accurate - the point was that the machines were internally 
inaccurate. 

Ricki - And it was too slow 

Don - Also it was too slow and we speeded it up for technical purposes 
from six inches ppr minute to 10^ inches per minute and that^s what 
the final speed will be. 

Ricki - Because when there was lots of activity on Idle film we were 
writing letters 

Don - They would pile up on each other 

Ricki - And another thing we found we were doing was that if that paper 
wasn't exactly right you had to watch it and if you were watching it, 
you weren't watching the film and it was moving so slowly that we had 
to watch the paper so we wouldn't mark letters on top of each other. 

? - Are our new machines going to have a little bitty space where 

we can write letters? 

Don - Uh-huh - a little bitty space - it will be a box - you may have 
seen the thing down there - it's about this wide and has legs on it 
and the machine itself will be one long table eind the four raters 
will just move their chairs up to this table and the little hole here 
for this rater and one position and there are four positions on the 
tape and one piece of adding machine tape is fed by motor-driven, . . 
and it just runs along this little tray here, so that everybody is 
rating on the same sheet of paper so that this means that synchroniza- 
tion is perfect; you can't miss. 



Rlcki - If one pencdLL slows down the paper it slows down for every- 
body. 

Don - It slows all of them down, but Jim is working out the kind of 
drive mechanization where paper will tear before it will slow down. 
Laughter... 

Don - This has been fascinating to me to see how all of these cognitive 
things that we have evolved reflected in the kind of machine we 
ended up with. We started out with the content that the deal was 
on a solenoid'^which wotild be moving four little pieces of paper for- 
ward every few seconds, and now we have ended up with one band of 

paper where four raters rate simultaneously. 

♦ 

Lynn - I don’t believe we have ever said why it was we needed the 
synchronization. 

Don - Oh, that’s your department. 

Lynn - Because we get these four nice long sheets of paper and try 
to figure out what responses came at the same or came to the different 
raters from the same stimulus and you couldn’t do it. You could get 
different kinds of ways to score this thing, then you might as well 
know for sure what it is you are working for. 

Ricki - And one of the ways we tried to fix them for you was for 
someone to call ’’Mark” every so often. And finally got down to a 
’’beep” every ten seconds, but that still didn’t help too much with 
the tapes . 

Pat - I think it is very Important. . .sticking to this idea of concen- 
trating on just one thing. 

Don - Yeah, this is what. ..this machine has finally evolved put so 
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tiiat: it can do this job. • .all you will have to do is two things: look 

at the film and sit there and write letters without looking. They 
will be in the right spot pmd they will be synchronized with every- 
body else. 

^ *• That reminds me of something else, too... you know those machines 
we had at first that were rectangular and long. ..they took up so 
much room on top cf the desk that they interfered with plugging the 
earphones into the projector. 

Don - Urrihum. 

Sheila - I liked the little ones so much better because that room is so 
cramped. I didn’t like all those wires. If you could avoid the 
wires. 

Ricki - We need a bigger room; we really do. 

Don - Incidentally, the earphones now when the final machine is 
finished. . .you will have your plug right in front of you, and this 
thing has a tray for all of your pencils and papers and your ear- 
phones plug them right in; tiiey don’t have to be unwired every time 
you walk off. 

Jeanne - I still don’t understand what the paper is going to look 
like when the rating is over... you pull out this one big sheet of 
paper and what is it going to look like? 

Ricki - One rater starts here and here and the next one here. 

Lynn - You have four rows on one sheet of paper, and what you do 
will split it and make... 

Ricki - •• .long-way. . . 

Jeanne - And you will just match them up? 
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Don - You will just match the starting marks, and liien everything 
following has to be synchronized. 

? - That^s going to be a J,ot of trouble. 

Ricki - Well, I was thinking the best way to cut them would be to 
line them up, and Idien you could have a razor blade and run it back 
through again. 

Don - You get a little old crank and four razor blades and crank it 
through there. . .take. . .you maybe, iiA fact you don^t even have to have 
a crank; you can just have a little gadget with a little lid over 
there and drop the paper in and throw the lid down and zip the paper 
through there, and you’ve got four strips. 

Lynn - You’re sure going to have to have sharp razor blades. 

Ricki - You sure will! 

Various jokes... 

Ricki - Is that it? Are we up to date? 

Don - I think it would be good to point out that so far in this we 
have not rated one official film. 

? - You had ta contaminate this. 

Ricki - Well, we tried. We con^letely rated one film, TAP and... 

Don - Oh, we’ve done it... 

Ricki - And it was a real film, and we got it together, and the 
machines weren’t synchronized, and Lynn couldn’t work out the relia- 
bility... 

Don - Yeah, we did that, too, but what I am saying that in spite of 
the fact that we are ri^t at the point where we feel like we could 
do it, we have never actually finished. 
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Ricki - We have never gotten the data back. 

Everyone talking at once, and laughter. •• 

Don - Speaking of hostile feelings, though, I reroeniber that when we 
did that one. ..well,. Lynn, this had better work, and she said it ‘- 
wouldn’t, and I thought. .. 

Laughter. . . 

Ricki - Well, we did get our reliability back on our TAF*s... 

Don - Oh, yeah. 

Ricki - Ours wasn^t tremendous, Don; I don'^t know about yours... 

Don - Well... 

Jeanne - Well, we changed ours; I would go a little bit more to the 
"three” if... 

Don - Our ratings... we had one there that was perfect agreement, and 
we had two and one or all together. . . 

Ricki - Well, we had four people on ours, and I think only five of the 
combinations came out significant... 

Don - I^bimim? 

Ricki - Hmmm? 

? - Yeah. 

Don - What are you using for significance? 

Lynn - .05 t. 

Ricki - And she just does it... 

Don - What were the coefficients? 

Ricki - Let's stop this thing first. 



